Ph.D. Dissertation Reliable Extraction of Realistic 3D Facial Animation Parameters from Mirror-reflected Multi-view video clips

With the rapid development of facial animation and facial motion analysis, the necessity of motion capture techniques increases dramatically. However, existing motion capture devices are still very expensive and have specific limitations. In this dissertation, an accurate and inexpensive procedure for estimating 3D facial and lip motion trajectories from mirror-reflected multi-view video is proposed. Two plane mirrors are located near a subject’s cheeks and a single digital video camcorder is utilized to capture markers’ front and side view images on a face simultaneously without special synchronization mechanisms. A novel closed-form linear algorithm is proposed to reconstruct 3D positions from real vs. mirrored point correspondences, where the extrinsic environment parameters do not need to be calibrated in advance. Since nice symmetric properties of mirrored objects are exploited, our computer simulations and expected error estimation manifest that the proposed 3D position estimation approach is more robust against noise, more accurate and simpler than general-purpose stereovision approaches by a linear algorithm or maximum likelihood optimization. In our experiments, a root mean square (RMS) error less than 2mm in 3D space can be reached while we use only 20 arbitrary point-corresponding pairs to evaluate the orientations and locations of mirror planes. For 3D facial motion extraction, our proposed procedure can track markers semi-automatically under normal light conditions. Adaptive Kalman predictors and filters are employed to improve the tracking stability and to conjecture the occluded markers’ positions. The motion tracking can be fully automatic with fluorescent markers illuminated by ultraviolet(UV) “blacklight blue”(BLB) lamps. For the problems of missing marker and false marker detection as well as false tracking, we employ the spatial coherence on face surfaces and the temporal coherence in motion to judge, rectify and compensate false tracking trajectories automatically. More than 300 markers on a subject’s face and lips are tracked from 30 fps video clips. This system will be extended for real-time tracking from live video in the near future. The estimated 3D facial motion data have also been practically applied to our facial animation system. In addition, a web-enabled talking head is also proposed, where facial animation is driven by natural speech. A speech analysis module is employed to obtain the corresponding phoneme sequence within the input speech, and then they are converted to the MPEG-4 high-level facial animation parameters called visemes to drive a 3D head model performing corresponding facial expressions. The talking head has been developed as plug-ins for web browsers and requires only 6 Kbps to stream high-resolution animation through Internet. Furthermore, my work was also used in a collaborative project between INRIA of France and National Taiwan University for a French-driven talking head system.